Workload Dependent Hadoop MapReduce Application Performance Modeling

In any distributed computing environment, performance optimization, job runtime predictions, or capacity and scalability quantification studies are considered as being rather complex, time-consuming and expensive while the results are normally rather error-prone. Based on the nature of the Hadoop MapReduce framework, many MapReduce production applications are executed against varying data-set sizes [5]. Hence, one pressing question of any Hadoop MapReduce setup is how to quantify the job completion time based on the varying data-set sizes and the physical and logical cluster resources at hand. Further, if the job completion time is not meeting the goals and objectives, does any Hadoop tuning or cluster resource adjustments result into altering the job execution time to actually meet the required SLA's.